Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Diabetic Nephropathy Screening in a Residency Clinic: Improvements Also Reveal Limits of Healthcare Effectiveness Data and Information Set (HEDIS) Metric.

Quality Improvement Success Stories are published by the American Diabetes Association in collaboration with the American College of Physicians, Inc., and the National Diabetes Education Program. This series is intended to highlight best practices and strategies from programs and clinics that have successfully improved the quality of care for people with diabetes or related conditions. Each article in the series is reviewed and follows a standard format developed by the editors of Clinical Diabetes. The following article describes a resident-led initiative that improved diabetic nephropathy screening in a primary care clinic. It also highlights the challenges of complex metrics, as well as the potential unintended consequences of emphasizing one dimension of a care process over another.

Integrated displays to improve chronic disease management in ambulatory care: A SMART on FHIR application informed by mixed-methods user testing.

OBJECTIVE: The study sought to evaluate a novel electronic health record (EHR) add-on application for chronic disease management that uses an integrated display to decrease user cognitive load, improve efficiency, and support clinical decision making. MATERIALS AND METHODS: We designed a chronic disease management application using the technology framework known as SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources). We used mixed methods to obtain user feedback on a prototype to support ambulatory providers managing chronic obstructive pulmonary disease. Each participant managed 2 patient scenarios using the regular EHR with and without access to our prototype in block-randomized order. The primary outcome was the percentage of expert-recommended ideal care tasks completed. Timing, keyboard and mouse use, and participant surveys were also collected. User experiences were captured using a retrospective think-aloud interview analyzed by concept coding. RESULTS: With our prototype, the 13 participants completed more recommended care (81% vs 48%; P < .001) and recommended tasks per minute (0.8 vs 0.6; P = .03) over longer sessions (7.0 minutes vs 5.4 minutes; P = .006). Keystrokes per task were lower with the prototype (6 vs 18; P < .001). Qualitative themes elicited included the desire for reliable presentation of information which matches participants' mental models of disease and for intuitive navigation in order to decrease cognitive load. DISCUSSION: Participants completed more recommended care by taking more time when using our prototype. Interviews identified a tension between using the inefficient but familiar EHR vs learning to use our novel prototype. Concept coding of user feedback generated actionable insights. CONCLUSIONS: Mixed methods can support the design and evaluation of SMART on FHIR EHR add-on applications by enhancing understanding of the user experience.